1. Field of the Invention
The present invention relates generally to an image processing system, and more particularly to the use of a generic image processing system to detect drowsiness.
2. Description of the Related Art
It is well known that a significant number of highway accidents result from drivers becoming drowsy or falling asleep, which results in many deaths and injuries. Drowsiness is also a problem in other fields, such as for airline pilots and power plant operators, in which great damage may result from failure to stay alert
A number of different physical criteria may be used to establish when a person is drowsy, including a change in the duration and interval of eye blinking. Normally, the duration of bilking is about 100 to 200 ms when awake and about 500 to 800 ms when drowsy. The time interval between successive blinks is generally constant while awake, but varies within a relatively broad range when drowsy.
Numerous devices have been proposed to detect drowsiness of drivers. Such devices are shown, for example, in U.S. Pat. Nos. 5,841,354; 5,813,993; 5,689,241; 5,684,461; 5,682,144; 5,469,143; 5,402,109; 5,353,013; 5,195,606; 4,928,090; 4,555,697, 4,485,375; and 4,259,665. In general, these devices fall into three categories: i) devices that detect movement of the head of the driver, e.g., tilting; ii) devices that detect a physiological change in the driver, e.g., altered heartbeat or brething, and iii) divice that detect a physical result of the driver falling asleep, e.g., a reduced grip of the steering wheel. None of this divice is believed to have met with commercial success.
The German patent application DE 19715519 and the corresponding French patent application FR-2.747.346 disclose an apparatus and process of evaluation of the drowsiness level of a driver using a video camera placed near the feet of the driver and a processing unit for processing the camera image with software detecting the blinks of the eyes to determine the time gap between the beginning and the end of the blinks. More particularly, a unit of the processor realizes
a memorization of the video image and its treatment, so as so determine an area comprising the driver""s eyes,
the dectetion of the time gap between the closing oft he driver ayelids and their full opening and
a treatment in a memo and a processor in combination with the unit to calculate a ratio of slow blink apparition.
The object of the international patent application published WO-97/01246 is a security system comprising a video camera placed within the rear-view mirror of a car and a video screen remotely disposed for the analysis of what is happening in the car and around it, as well as of what happened due to the recording of the output video signal of the camera This is in fact a concealed camera (within the rear-view mirror), so that it is imperceptible to vandals and thieves and which observes a large scope including the inside of the car and its surroundings, the record allowing one to know later what has happened in this scope (page 6, lines 13 to 19), this is not a detector whose effective angle is strictly limited to the car driver face in order to detect its eventual drowsiness and to make him awake.
Commonly-owned PCT Application Ser. Nos. PCT/FR97/01354 and PCT/EP98/05383 disclose a generic image processing system that operates to localize objects in relative movement in an image and to determine the speed and direction of the objects in real-time. Each pixel of an image is smoothed using its own time constant. A binary value corresponding to the existence of a significant variation in the amplitude of the smoothed pixel from the prior frame, and the amplitude of the variation, are determined, and the time constant for the pixel is updated. For each particular pixel, two matrices are formed that include a subset of the pixels spatially related to the particular pixel. The first matrix contains the binary values of the subset of pixels. The second matrix contains the amplitude of the variation of the subset of pixels. In the first matrix, it is determined whether the pixels along an oriented direction relative to the particular pixel have binary values representative of significant variation, and, for such pixels, it is determined in the second matrix whether the amplitude of these pixels varies in a known manner indicating movement in the oriented direction. In domains that include luminance, hue, saturation, speed, oriented direction, time constant, and x and y position, a histogram is formed of the values in the first and second matrices falling in user selected combinations of such domains. Using the histograms, it is determined whether there is an area having the characteristics of the selected combinations of domains.
It would be desirable to apply such a generic image processing system to detect the drowsiness of a person.
The present invention is a process of detecting a driver falling asleep in which an image of the face of the driver is acquired. Pixels of the image having characteristics corresponding to characteristics of at least one eye of the driver are selected and a histogram is formed of the selected pixels. The histogram is analyzed over time to identify each opening and closing of the eye, and from the eye opening and closing information, characteristics indicative of a driver falling asleep are determined.
In one embodiment, a sub-area of the image comprising the eye is determined prior to the step of selecting pixels of the image having characteristics corresponding to characteristics of an eye. In this embodiment, the step of selecting pixels of the image having characteristics of an eye involves selecting pixels within the sub-area of the image. The step of identifying a sub-area of the image preferably involves identifying the head of the driver, or a facial characteristic of the driver, such as the driver""s nostrils, and then identifying the sub-area of the image using an anthropomorphic model. The head of the driver may be identified by selecting pixels of the image having characteristics corresponding to edges of the head of the driver. Histograms of the selected pixels of the edges of the driver""s head are projected onto orthogonal axes. These histograms are then analyzed to identify the edges of the driver""s head.
The facial characteristic of the driver may be identified by selecting pixels of the image having characteristics corresponding to the facial characteristic. Histograms of the selected pixels of the facial characteristic are projected onto orthogonal axes. These histograms are then analyzed to identify the facial characteristic. If desired, the step of identifying the facial characteristic in the image involves searching sub-images of the image until the facial characteristic is found. In the case in which the facial characteristic is the nostrils of the driver, a histogram is formed of pixels having low luminance levels to detect the nostrils. To confirm detection of the nostrils, the histograms of the nostril pixels may be analyzed to determine whether the spacing between the nostrils is within a desired range and whether the dimensions of the nostrils fall within a desired range. In order to confirm the identification of the facial characteristic, an anthropomorphic model and the location of the facial characteristic are used to select a sub-area of the image containing a second facial characteristic. Pixels of the image having characteristics corresponding to the second facial characteristic are selected and histograms of the selected pixels of the second facial characteristic are analyzed to confirm the identification of the first facial characteristic.
In order to determine openings and closings of the eyes of the driver, the step of selecting pixels of the image having characteristics corresponding to characteristics of an eye of the driver involves selecting pixels having low luminance levels corresponding to shadowing of the eye. In this embodiment, the step of analyzing the histogram over time to identify each opening and closing of the eye involves analyzing the shape of the eye shadowing to determine openings and closings of the eye. The histograms of shadowed pixels are preferably projected onto orthogonal axes, and the step of analyzing the shape of the eye shadowing involves analyzing the width and height of the shadowing.
An alternative method of determining openings and closings of the eyes of the driver involves selecting pixels of the image having characteristics of movement corresponding to blinking. In this embodiment, the step of analyzing the histogram over time to identify each opening and closing of the eye involves analyzing the number of pixels in movement corresponding to blinking over time. The characteristics of a blinking eye are preferably selected from the group consisting of i) DP=1, ii) CO indicative of a blinking eyelid, iii) velocity indicative of a blinking eyelid, and iv) up and down movement indicative of a blinking eyelid.
An apparatus for detecting a driver falling asleep includes a sensor for acquiring an image of the face of the driver, a controller, and a histogram formation unit for forming a histogram on pixels having selected characteristics. The controller controls the histogram formation unit to select pixels of the image having characteristics corresponding to characteristics of at least one eye of the driver and to form a histogram of the selected pixels. The controller analyzes the histogram over time to identifies each opening and closing of the eye, and determines from the opening and closing information on the eye, characteristics indicative of the driver falling asleep.
In one embodiment, the controller interacts with the histogram formation unit to identify a sub-area of the image comprising the eye, and the controller controls the histogram formation unit to select pixels of the image having characteristics corresponding to characteristics of the eye only within the sub-area of the image. In order to select the sub-area of the image, the controller interacts with the histogram formation unit to identify the head of the driver in the image, or a facial characteristic of the driver, such as the driver""s nostrils. The controller then identifies the sub-area of the image using an anthropomorphic model. To identify the head of the driver, the histogram formation unit selects pixels of the image having characteristics corresponding to edges of the head of the driver and forms histograms of the selected pixels projected onto orthogonal axes. To identify a facial characteristic of the driver, the histogram formation unit selects pixels of the image having characteristics corresponding to the facial characteristic and forms histograms of the selected pixels projected onto orthogonal axes. The controller then analyzes the histograms of the selected pixels to identify the edges of the head of the driver or the facial characteristic, as the case may be. If the facial characteristic is the nostrils of the driver, the histogram formation unit selects pixels of the image having low luminance levels corresponding to the luminance level of the nostrils. The controller may also analyze the histograms of the nostril pixels to determine whether the spacing between the nostrils is within a desired range and whether dimensions of the nostrils fall within a desired range. If desired, the controller may interact with the histogram formation unit to search sub-images of the image to identify the facial characteristic.
In order to verify identification of the facial characteristic, the controller uses an anthropomorphic model and the location of the facial characteristic to cause the histogram formation unit to select a sub-area of the image containing a second facial characteristic. The histogram formation unit selects pixels of the image in the sub-area having characteristics corresponding to the second facial characteristic and forms a histogram of such pixels. The controller then analyzes the histogram of the selected pixels corresponding to the second facial characteristic to identify the second facial characteristic and to thereby confirm the identification of the first facial characteristic.
In one embodiment, the histogram formation unit selects pixels of the image having low luminance levels corresponding to shadowing of the eyes, and the controller then analyzes the shape of the eye shadowing to identify shapes corresponding to openings and closings of the eye. The histogram formation unit preferably forms histograms of the shadowed pixels of the eye projected onto orthogonal axes, and the controller analyzes the width and height of the shadowing to determine openings and closings of the eye.
In an alternative embodiment, the histogram formation unit selects pixels of the image in movement corresponding to blinking and the controller analyzes the number of pixels in movement over time to determine openings and closings of the eye. The characteristics of movement corresponding to blinking are preferably selected from the group consisting of i) DP=1, ii) CO indicative of a blinking eyelid, iii) velocity indicative of a blinking eyelid, and iv) up and down movement indicative of a blinking eyelid.
If desired, the sensor may be integrally constructed with the controller and the histogram formation unit. The apparatus may comprise an alarm, which the controller operates upon detection of the driver falling asleep, and may comprise an illumination source, such as a source of IR radiation, with the sensor being adapted to view the driver when illuminated by the illumination source.
A rear-view mirror assembly comprises a rear-view mirror and the described apparatus for detecting driver drowsiness mounted to the rear-view mirror. In one embodiment, a bracket attaches the apparatus to the rear-view mirror. In an alternative embodiment, the rear-view mirror comprises a housing having an open side and an interior. The rear-view mirror is mounted to the open side of the housing, and is see-through from the interior of the housing to the exterior of the housing. The drowsiness detection apparatus is mounted interior to the housing with the sensor directed toward the rear-view mirror. If desired, a joint attaches the apparatus to the rear-view mirror assembly, with the joint being adapted to maintain the apparatus in a position facing the driver during adjustment of the mirror assembly by the driver. The rear-view mirror assembly may include a source of illumination directed toward the driver, with the sensor adapted to view the driver when illuminated by the source of illumination. The rear-view mirror assembly may also include an alarm, with the controller operating the alarm upon detection of the driver falling asleep. Also disclosed is a vehicle comprising the drowsiness detection device.